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~ The MAILING DATE of this communication appears on the cover sheet with the correspondence address- 

All claims being allowable, PROSECUTION ON THE MERITS IS (OR REMAINS) CLOSED in this application. If not included 
herewith (or previously mailed), a Notice of Allowance (PTOL-85) or other appropriate communication will be mailed in due course. THIS 
NOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the initiative 
of the Office or upon petition by the applicant. See 37 CFR 1.313 and MPEP 1308. 

1 . |EI This communication is responsive to an amendment filed on 6/27/2008. 

2. ^ The allowed claim(s) is/are 9-16 . 

3. ^ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a) ^All b)DSome* c) □ None of the: 

1. □ Certified copies of the priority documents have been received. 

2. □ Certified copies of the priority documents have been received in Application No. . 

3. □ Copies of the certified copies of the priority documents have been received in this national stage application from the 

International Bureau (PCT Rule 17.2(a)). 
* Certified copies not received: . 

Applicant has THREE MONTHS FROM THE "MAILING DATE" of this communication to file a reply complying with the requirements 
noted below. Failure to timely comply will result in ABANDONMENT of this application. 
THIS THREE-MONTH PERIOD IS NOT EXTENDABLE. 

4. □ A SUBSTITUTE OATH OR DECLARATION must be submitted. Note the attached EXAMINER'S AMENDMENT or NOTICE OF 

INFORMAL PATENT APPLICATION (PTO-152) which gives reason(s) why the oath or declaration is deficient. 

5. □ CORRECTED DRAWINGS ( as "replacement sheets") must be submitted. 

(a) □ including changes required by the Notice of Draftsperson's Patent Drawing Review ( PTO-948) attached 

1 ) □ hereto or 2) □ to Paper No./Mail Date . 

(b) □ including changes required by the attached Examiner's Amendment / Comment or in the Office action of 

Paper No./Mail Date . 

Identifying indicia such as the application number (see 37 CFR 1. 84(c)) should be written on the drawings in the front (not the back) of 
each sheet. Replacement sheet(s) should be labeled as such in the header according to 37 CFR 1.121(d). 
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EXAMINER'S AMENDMENT 

1. An examiner's amendment to the record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1.312. To ensure consideration of such an amendment, it MUST be 
submitted no later than the payment of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview 
with Kevin McDermott on Tuesday, August 13 2008. 

The Examiner has made the following changes to the claims. Additions to the 
claims are reflected by underline ( example ) and deletions are reflected by strikethrough 
( examp l e ). 

Claim 9 

An ingredient cooking-operation recognition system comprising: 

a sensing means for acquiring observation data which includes an image around 

the hands of a person who performs a cooking operation and an environment sound 

that is produced by the cooking operation; 

a feature-quantity template in which various cooking operations that are 

predetermined for various ingredients are stored together with a template certainty 

factor that is a certainty factor which is predetermined for each cooking operation of the 

ingredients; 

a cooking-flow creating means for, based on the observation data, calculating an 
observation certainty factor which indicates the certainty factor of at least either of an 
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ingredient that is cooked by the person and the cooking operation of the ingredient, and 
based on the observation certainty factor, creating a cooking flow for a dish which is 
prepared by the person; 

a primary certainty-factor calculating means for, calculating a primary certainty 
factor which is a certainty factor for each object that makes up the cooking flow based 
on the template certainty factor and the observation certainty factor; 

a cooking-operation recognizing means for recognizing the ingredient and the 
cooking operation that correspond to the observation data, based on the primary 
certainty factor; and 

a guidance means for providing guidance to the person regarding the cooking 
r e c i p e flow based on the ingredient and the cooking operation recognized by the 
cooking-operation recognizing means, wherein the person performs the cooking 
operation in accordance with the guidance. 

Claim 16 

A computer-readable recording medium which has recorded thereon an 
ingredient cooking-operation recognition program, for causing a computer to function at 
least as: 

a sensing means for acquiring observation data which includes an image around 
the hands of a person who performs a cooking operation and an environment sound 
that is produced by the cooking operation; 
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a feature-quantity template in which various cooking operations that are 
predetermined for various ingredients are stored together with a template certainty 
factor that is a certainty factor which is predetermined for each cooking operation of the 
ingredients; 

a cooking-flow creating means for, based on the observation data, calculating an 
observation certainty factor which indicates the certainty factor of at least either of an 
ingredient that is cooked by the person and the cooking operation of the 
ingredient, and based on the observation certainty factor, creating a cooking flow for a 
dish which is prepared by the person; 

a primary certainty-factor calculating means for, calculating a primary certainty 
factor which is a certainty factor for each object that makes up the cooking flow based 
on the template certainty factor and the observation certainty factor; and 

a cooking-operation recognizing means for recognizing the ingredient and the 
cooking operation that correspond to the observation data, based on the primary 
certainty factor; and 

a guidance means for providing guidance to the person regarding the cooking 
r e c i p e flow based on the ingredient and the cooking operation recognized by the 
cooking-operation recognizing means, wherein the person performs the cooking 
operation in accordance with the guidance 
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Reasons for Allowance 

2. The following is an examiner's statement of reasons for allowance: claims 9-16 
are considered allowable since when reading the claims in light of the specification, as 
per MPEP § 2111.01, In re Donaldson Co.. Inc.. 29 USPQ 2d 1845, 1850 (Fed. Cir. 
1994), none of the references of record alone or in combination disclose or suggest the 
combination of limitations specified in the independent claims, specifically a feature 
quantity template storing various ingredients together with a template certainty factor 
(as defined at e.g. page 2, L7-17; pg. 4, L1-5; pg. 9, L4 to pg. 12, L21; pg. 16 L20 to pg. 
18, L26; Figs. 4, 6 of the specification of the instant application), calculating an 
observation certainty factor used to create a cooking flow for an ingredient being cooked 
or a cooking operation being performed (as defined at e.g. pg. 19, L1-18 of the 
specification of the instant application) and a primary certainty factor for each object that 
makes up the cooking flow based on the template certainty factor and the observation 
certainty factor (as defined at e.g. pg. 20, L3 to pg. 21, L1). 

Hamada et al. (Associating Cooking Video with Related textbook) discloses a 
system that integrates analysis of an image, audio and text and associate each other. 
Text data is analyzed to extract keywords. Video and audio are analyzed and 
recombined in semantic scenes. The text and the semantic scenes of the video are 
associated and a cooking recipe is created. 

However, Hamada does not teach the combination of limitations specified in 
independent claims 9 and 16, specifically a feature quantity template storing various 
ingredients together with a template certainty factor, calculating an observation certainty 
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factor used to create a cooking flow for an ingredient being cooked or a cooking 
operation being performed and a primary certainty factor for each object that makes up 
the cooking flow based on the template certainty factor and the observation certainty 
factor. 

Conclusion 

3. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Miura et al. "Associating Semantically Structured Cooking Videos with their 
Preparation Steps" System and Computers in Japan, 2005, pages 51-62. 

Tee et al. "A Visual Recipe Book for Persons with Language Impairments" CHI 
2005, April 2-7 2005, pages 501-510. 

4. Claims 9-16 are allowed. 

Correspondence Information 

5. Any inquires concerning this communication or earlier communications from the 
examiner should be directed to Omar F. Fernandez Rivas, who may be reached 
Monday through Friday, between 8:00 a.m. and 5:00 p.m. EST. or via telephone at 
(571) 272-2589 or email omar.fernandezrivas@uspto.gov. 

If you need to send an Official facsimile transmission, please send it to (571) 
273-8300. 
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If attempts to reach the examiner are unsuccessful the Examiner's Supervisor, 
David Vincent, may be reached at (571 ) 272-3080. 

Hand-delivered responses should be delivered to the Receptionist @ (Customer 
Service Window Randolph Building 401 Dulany Street Alexandria, VA 22313), located 
on the first floor of the south side of the Randolph Building. 
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Patent Examiner 
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Patent & Trademark Office 

/Omar F. Fernandez Rivas/ 
Examiner, Art Unit 2129 

Wednesday, August 13, 2008. 
/David R Vincent/ 

Supervisory Patent Examiner, Art Unit 2129 



